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Historical  Lessons  of  High  Costs 

This  is  not  a  new  phenomenon  to  DoD.  Weap¬ 
ons  acquisition  throughout  U.S.  history  experi¬ 
enced  numerous  cost-estimate  errors.  Among 
the  first  contractors  bedeviled  by  cost  prob¬ 
lems  was  Joshua  Humphreys,  the  creator  of 
the  Navy’s  first  six  frigates.  These  ships  were 
designed  to  be  heavily  armed  and  able  to  out- 
gun  any  European  ship  of  a  similar  class,  yet 
fast  enough  to  outrun  any  larger  ship  in  a  light 
breeze.  The  big  frigates  had  special  construc¬ 
tion  requirements  that  required  live  oak  timber 
for  critical  components.  According  to  Ian  W. 
Toll,  author  of  Six  Frigates:  The  Epic  History  of 
the  Founding  of  the  U.S.  Navy,  several  hundred 
live  oak  trees  were  needed  for  each  ship. 

There  was  a  problem,  though.  Live  oak  grows 
best  in  the  coastal  plains  of  the  southeastern 
United  States.  At  the  end  of  the  1 8th  century 
and  the  beginning  of  the  1 9th  century,  har¬ 
vesting  the  enormous  live  oak  trees  was  a 
painstakingly  slow  and  expensive  proposition 
because  the  coastal  plains  had  few  roads  and 
an  abundance  of  disease-carrying  mosquitoes. 
So  expensive,  in  fact,  that  in  1794,  the  U.S.  House  of 
Representatives  appointed  a  special  committee  to  inves¬ 
tigate  how  $7,000  could  be  spent  on  timber  in  a  single 
month,  especially  when  the  estimated  cost  was  much 
lower.  Pointed  queries  were  made,  egos  were  wounded, 
and  political  posturing  abounded,  but  the  program  edged 
on  to  a  satisfactory  completion. 

Reasons  for  Cost-Estimating  Problems 

Looking  at  this  example  and  numerous  other  occasions  in 
DoD’s  history  in  which  new  weapons  systems  resulted  in 
significant  underestimations  of  costs,  three  factors  appear 
to  be  associated  with  costing  errors: 

■  A  new  technology  or  concept  is  introduced.  In  Joshua 
Humphreys’  case,  the  innovation  was  the  design  of  the 
ship  and  the  materials  required. 


Department  of  Defense  weapons  systems  and  pro¬ 
gram  developments  have  received  considerable 
unwanted  attention,  not  because  of  innovative 
design  issues,  but  rather,  because  rising  costs 
concern  those  who  control  the  money.  In  fact, 
cost  overruns  in  some  programs  have  pushed  the  price 
well  beyond  the  original  estimates. 
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Figure  1 .  Data  Construct 
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“Give  me  a  ROM  in  two  weeks 
if  you  really  want  any  chance 
of  funding  this  initiative,”  is  the 
reply. 


Changes  in  design  after  the  system  is  in  production. 
These  changes  invariably  result  in  unintended  conse¬ 
quences  and  additional  costs. 

The  contractor  routinely  accepts  that  the  technology  is 
mature  and  there  will  be  no  design  modifications  after 
the  system  is  produced.  This  results  in  an  initial  low-cost 
estimate  that  is  very  often  unrealistic. 


The  project  engineer  walks  away  thinking, 
“How  am  I  going  to  develop  a  sound,  engineering- 
based  cost  estimate  in  two  weeks?”  Of  course,  the  project 
engineer  is  correct.  It  will  take  several  months,  or  even 
longer,  to  develop  a  realistic  cost  model  of  a  complex 
system  or  major  system  upgrade,  as  well  as  the  logis¬ 
tics  tail  associated  with  any  change  to  a  fielded  system 
(spares,  technical  manuals,  allowance  parts  lists,  planned 
maintenance  systems,  integrated  logistics  support  plans, 
and  so  on). 


At  the  beginning  of  the  21  st  century,  problems  with  cost- 
estimate  accuracy,  a  history  of  cost  growth,  and  high  vis¬ 
ibility  have  caused  DoD  program  managers  to  seek  more 
detailed  cost  estimates  with  frequent  updates.  However, 
manic  attention  to  cost  estimating  also  has  its  price.  Pro¬ 
gram  management  and  engineering  personnel  throughout 
the  defense  industry  expend  time,  energy,  and  resources 
in  developing,  validating,  and  certifying  cost  estimates. 
The  unplanned  workload  and  reporting  requirements  are 
placing  a  strain  on  an  overburdened  logistics  system.  In 
many  cases,  programs  simply  have  inadequate  staffs  to 
manage  the  contracts. 

Not  Enough  Time 

There  is  a  growing  need  to  develop  cost  estimates  to  sup¬ 
port  a  variety  of  managerial,  programmatic,  and  engineer¬ 
ing  requirements,  and  get  it  done  fast.  That’s  another 
cost-estimating  problem.  Routinely,  the  technical  commu¬ 
nity  receives  a  proposal  to  improve  a  weapons  system’s 
performance  to  meet  emergent  requirements.  Once  the 
proposal  is  briefed  to  the  chain  of  command,  the  project 
engineer  is  asked,  “How  much  will  it  cost?” 


The  related  scenario  is  typical  of  situations  encountered 
daily  in  defense  industry  engineering  efforts.  A  tool  is 
needed  to  enable  a  robust,  engineering-based  modeling 
and  simulation  of  system-level  technical  characteristics, 
including  the  required  performance  parameters  and  as¬ 
sociated  costs.  It  must  be  an  automated  tool  that  helps 
streamline  the  existing  laborious  process  of  collecting 
component  data  and  projecting  technical  cost  for  perfor¬ 
mance  trades,  and  that  assists  in  determining  schedule 
considerations  and  technology  maturity. 

The  Tools  Are  Out  There 

Many  of  the  elements  of  the  M&S  system  are  read¬ 
ily  available.  There  is  an  abundance  of  technical  data 
available  from  component  manufacturers  and  govern¬ 
ment  research  activities.  The  project  engineer’s  vision 
can  provide  the  required  system  parameters.  With  the 
data  and  expertise  in  place,  all  that  is  left  is  to  establish 
the  statistical  relationships  between  the  data,  weigh  the 
relative  importance  of  the  data,  and  possibly  develop  a 
qualitative  ranking  schema.  Thus,  the  resulting  estimate 
will  be  based  on  historical  and  factual  data,  coupled  with 
projections  of  new  developments  and  price  points  from 
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characteristics  of  a  system  to  an  element  of  its  cost. 
An  example  of  a  parametric  cost  estimate  may  be  an 
estimate  based  on  the  system’s  weight  or  the  space  it 
will  occupy  on  board  the  ship. 

■  Analogy  cost  estimation  compares  the  proposed  sys¬ 
tem  to  an  existing  system  that  is  similar  in  design  or 
operation.  For  example,  a  proposed  radar  system  may 
be  evaluated  against  an  existing  similar  radar  system. 

■  Engineering  cost  estimates  are  based  on  detailed  bot- 
toms-up  calculations  and  are  the  most  time-consuming 
of  the  three  techniques.  Extensive  amounts  of  detailed 
data  and  labor  hours  are  required  for  this  approach, 
and  still  the  quality  of  the  estimates  is  dependent  on 
the  credibility  of  the  data  available. 

Modeling  and  Simulation-Based  Solution 

The  ideal  M&S-based  cost-estimating  system  will  use  all 
three  of  the  aforementioned  cost-estimating  methods, 
blending  data  stores  of  parametric  data,  existing  systems 
costs  (historical  data),  and  detailed  cost  and  technical  data 
at  the  component  level. 


research  activities  and  with  the  project  engineer’s  vision 
for  the  desired  architecture. 

The  algorithms  and  statistical  modeling  as  well  as  the 
complex  relational  database  storage  and  retrieval  systems 
have  been  developed  for  other  DoD  applications  over  the 
past  five  years.  It  is  feasible  that  a  system-level  M&S  tool 
can  be  developed  to  actually  model  system-level  perfor¬ 
mance  and  associated  costs  and  schedule  considerations. 
Taking  advantage  of  the  DoD  tools  available  will  allow  the 
creation  of  an  M&S  cost-estimating  system  that  can  easily 
develop  complex  estimates  in  a  timely  manner. 

Once  the  base  estimate  is  developed,  the  project  engineer 
and  program  personnel  can  evaluate  “what  ifs”  by  varying 
the  performance  characteristics,  thus  testing  the  system’s 
maximum  performance  potential  with  minimal  cost  and 
schedule  factors.  Performing  the  test  in  a  digital  data  en¬ 
vironment  will  create  sufficient  rigor  in  the  estimates  and 
will  provide  the  ability  to  quickly  and  efficiently  develop 
various  scenarios  with  the  data— including  system-per¬ 
formance  characteristics— resulting  in  a  comprehensive 
system-level  cost  estimate. 

Current  Cost-Estimating  Methodologies 

Typically,  cost  estimates  are  developed  by  the  selection 
of  one  or  more  estimating  methods.  The  methods  are 
usually  determined  by  the  amount  and  quality  of  avail¬ 
able  cost  data.  There  are  three  common  types  of  cost 
estimating: 

■  Parametric  cost  estimates  use  an  equation  to  represent 
the  cost-estimating  relationships  between  one  or  more 


A  preliminary  systems  design  can  be  created  based  on 
the  accumulated  data  and  on  a  series  of  customer  re¬ 
sponses.  The  weighting  of  each  of  the  models’  cost-es¬ 
timating  methods  will  be  adjusted  based  on  the  relative 
accuracy  score  of  each  specific  data  element.  The  result  is 
a  detailed,  composite  cost  estimate  based  on  all  available 
data  at  the  time  of  the  estimate.  To  validate  the  reliability 
of  the  model,  actual  historical  cost  data  will  be  used  to 
develop  a  cost  estimate,  and  then  historical  actual  cost 
data  will  be  analyzed  to  refine  the  ability  of  the  model  to 
accurately  predict  future  estimates.  As  actual  data  become 
available,  the  model  will  be  updated  automatically  to  en¬ 
sure  that  the  data  weighing  and  statistical  relationships 
are  optimized. 

Figure  1  is  the  proposed  data  construct  for  a  radio  fre¬ 
quency  system  components/modules  cost-estimating 
M&S  tool.  The  data  construct  shows  the  source  and  na¬ 
ture  of  the  information  to  be  collected  and  stored.  There 
is  a  pathway  of  two-way  communication  between  an 
M&S  cost-estimating  tool  and  all  of  the  key  data  hold¬ 
ers:  academia,  risk  management  analysis  data  stores, 
government  research  laboratories,  statisticians,  original 
equipment  manufacturers  and  system  developers,  cost 
estimators,  naval  inventory  control  point,  and  commercial 
radio  frequency  manufacturers. 

In  Figure  1 ,  a  series  of  technical  interchanges  is  conducted 
to  ensure  the  development  of  the  radio  frequency  system 
components/modules  cost-estimating  M&S  tool,  guided 
by  expert  knowledge  and  best  industry  practices.  This 
knowledge  is  captured  and  leveraged  via  Lean  Six  Sigma 
and  teaming  arrangements. 
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Figure  2.  Modified  Waterfall  Incremental 
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will  be  performed.  A  systems  engineering  approach  to  the 
cost-estimating  procedure  will  ensure  a  disciplined  meth¬ 
odology  based  on  proven  techniques.  This  process 
will  require  a  number  of  studies,  and  it  will  include 
analyzing  needs,  operational  requirements,  func- 
tions,  concept  of  operations,  and  performance. 


When  the  studies  are  complete  and  the  lessons 
learned,  conclusions  and  recommendations 
are  evaluated.  The  development  phase  will 
begin. 


Database 


A  combination  of  models  is  used  to  create  realistic  cost 
estimates  for  weapons  systems.  Cost-estimating  models 
used  in  the  weapons  system  cost-estimating  M&S  tool  will 
include  a  parametric  model,  an  accounting  model,  a  simu¬ 
lation  model,  and  statistical  simulations.  The  parametric 
model  will  contain  a  set  of  equations,  each  of  which  re¬ 
lates  costs  to  parameters  that  describe  the  design,  per¬ 
formance,  operating  characteristics,  or  operating  environ¬ 
ment  of  a  weapons  system.  An  accounting  model  will  be 
included  and  will  contain  a  set  of  equations  used  to  com¬ 
bine  elements  of  costs  from  simple  relationships  or  direct 
inputs.  Some  elements  will  be  computed  on  the  basis  of 
unit  cost  and  procurement  quantity  while  others  will  be 
estimated  using  separate  models  or  methodologies.  These 
estimates  are  provided  to  arrive  at  an  aggregate  estimate 
of  costs.  A  computer  simulation  model  will  determine  the 
effect  on  costs  of  a  system’s  characteristics,  operational 
constraints,  base  concept,  maintenance  plan,  and  spare 
and  support  requirements.  The  simulation  model  will 
break  costs  into  workable  elements  for  which  estimates 
are  then  developed  using  hardware  parameters  (such  as 
reliability,  maintainability,  etc.).  Statistical  simulations  will 
be  used  over  time  to  generate  probability  density  func¬ 
tions  that  describe  the  impact  of  system  characteristics, 
operations,  and  maintenance  concepts.  Because  of  the 
large  amount  of  data  required,  the  use  of  such  models 
is  normally  limited  to  the  later  program  phases  in  which 
sufficient  amounts  of  detailed  data  are  available. 

Putting  the  Tool  to  the  Test 

How  effective  is  the  M&S  tool  in  developing  complex  cost 
estimates?  Let’s  provide  a  proof-of-concept  demonstration 
in  which  we’ll  develop  ROM  cost  estimates  on  state-of-the- 
art  radio  frequency  technologies  to  be  inserted  into  exist¬ 
ing  and  new  radio  frequency  systems.  The  initial  focus  of 
this  M&S  effort  will  be  on  the  radio  frequency  sub-systems 
and  components  of  a  radio  frequency  system. 

To  develop  the  M&S  cost-estimating  system,  a  systems  en¬ 
gineering  development  analysis  and  initial  development 


If  the  initial  development  is  successful,  ad¬ 
ditional  system  modules  will  be  developed 
_  using  a  spiral  development,  or  a  modified 

^  waterfall  incremental  build  model,  de- 

Web  Access  picted  in  Figure  2.  The  actual  steps  in  each 
block  will  be  refined  based  on  input  and  con¬ 
sultation  with  team  members,  academia  staff, 
users,  program  office  representatives,  and  expert  data 
stores/management  personnel. 

Even  though  this  is  primarily  a  software  development 
effort,  the  principles  of  systems  engineering  still  apply. 
System  engineering  principles  and  methods  would  be 
applied  to  all  aspects  of  the  management  and  engineer¬ 
ing  development  phases  during  the  development  of  this 
project.  As  spiral  development  continues,  the  products 
will  be  built  and  improved  upon  to  provide  users  with  a 
more  robust,  accurate,  and  useful  M&S  cost-estimating 
tool.  Enforcing  the  rigors  of  systems  engineering,  techni¬ 
cal  reviews  will  be  conducted  on  the  following:  cost  model 
assumption,  data  manipulation  history,  and  the  statistical 
cost  curves  used  for  cost  elements  and  sub-elements.  The 
technical  reviews  will  be  conducted  by  peers  and  subject 
matter  experts. 

The  Changing  Face  of  Cost-Estimating 
Methods 

Both  historically  and  today,  DoD  is  facing  a  persistent 
problem  of  inaccurate  cost  estimates  based  on  outmoded 
and  inadequate  methods.  However,  data  and  techniques 
exist  that  can  improve  the  process  and  accuracy  of  weap¬ 
ons  systems  cost  estimation.  The  storage  of  all  cost-es¬ 
timate  models  will  allow  timely  updates  and  refinement 
of  the  overall  model.  The  alternative  is  to  continue  with 
the  current  cost-estimating  methods,  and  we  can  expect 
many  of  the  same  results  that  plagued  Joshua  Humphreys 
and  his  six  frigates  at  the  beginning  of  our  Navy’s  his¬ 
tory:  Cost  overruns,  missed  schedules,  pointed  inquiries, 
wounded  egos,  and  political  posturing  are  the  inevitable 
companions  of  poor  cost  estimates. 


The  authors  welcome  comments  and  questions 
and  can  be  contacted  at  james.m.young@navy. 
mil  and  ted.markley.ctr@navy.mil. 
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